Expert networks with mixed continuous and categorical feature variables: a location modeling approach

In the context of medically relevant artificial intelligence, many real-world problems involve both continuous and categorical feature variables. When the data are mixed mode, the assumption of multivariate Gaussian distributions for the gating network of normalized Gaussian (NG) expert networks, such as NG mixture of experts (NGME), becomes invalid. An independence model has been studied to handle mixed feature data within the framework of NG expert networks. This method is based on the NAIVE assumption that the categorical variables are independent of each other and of the continuous variables. While this method performs surprisingly well in practice as a way of handling problems with mixed feature variables, the independence assumption is likely to be unrealistic for many practical problems. In this chapter, we investigate a dependence model which allows for some dependence between the categorical and continuous variables by adopting a location modeling approach. We show how the expectation-maximization (EM) algorithm can still be adopted to train the location NG expert networks via the maximum likelihood (ML) approach. With the location model, the categorical variables are uniquely transformed to a single multinomial random variable with cells of distinct patterns (locations). Any associations between the original categorical variables are then converted into relationships among the resulting multinomial cell probabilities. In practice, the dependence model approach becomes intractable when the multinomial distribution replacing the categorical variables has many cells and/or there are many continuous feature variables. An efficient procedure is developed to determine the correlation structure between the categorical and continuous variables in order to minimize the number of parameters in the dependence model. The method is applied to classify cancer patients on the basis of continuous gene-expression-profile vector of tumour samples and categorical variables of patient’s clinical characteristics. The proposed methodologies would have wide application in various scientific fields such as economy, biomedical and health sciences, and many others, where data with mixed feature variables are collected. Further extensions of the methodologies to other NG networks and/or to other members of the exponential family of densities for the local output density are discussed.